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Executive summary 

Current economic conditions and the increasing competition for government funding are leading to an 
increased focus on the impact of research. Other factors contributing to this interest include the development 
of ERA which measures the academic excellence of research and the view that impact measures would 
complement this assessment; a general concern to improve the operation of the national innovation system; 
and the need to demonstrate to the public that research funds are well-spent. 

Measuring the impact of research is difficult because not all impacts are direct and some can be negative 
or result from the identification of problems that reguire a non-research response. The time between the 
performance of research and when its benefits become apparent can be significant, unpredictable and differ 
for different kinds of research. 

The likelihood of research having impact depends not just upon the potential of the research but also on the 
willingness and ability of players in the wider innovation system to make use of the research; and any research 
does not exist in isolation but draws on the work of other researchers. In assessing impact it is necessary to 
acknowledge that research aiming to achieve impact will often have a high risk of failure and that there can be 
different perspectives about whether a particular impact is positive or not. 

Attempts to measure impact, especially if these are ongoing, can distort behaviours such that they might 
diminish the probability of research reaching its maximum impact. A rigorous assessment of research impact 
has also to develop appropriate counterfactuals and consider opportunity costs. 

Research can have impact through many routes and in different ways. These range from building national 
capability through advancing knowledge and supporting university teaching, to producing a direct financial 
return to the institution performing the research and having major economic impact through increases in 
productivity, employment, competitiveness and business formation. Research can also contribute to national 
wellbeing through its social impacts and by improving environmental management and sustainability. There 
are also many intangible benefits of research which are nevertheless real and of value - including on national 
reputation and attractiveness as a place to learn, work and invest. 

There is a wide range of methods that it is possible to use to evaluate research impact. They can operate at 
the level of individual projects or programs, institutions and nations. Each method has its own characteristics 
and advantages. While different methods can appeal to different target groups, none is complete in itself and 
none offers unambiguous or certain results. Studies of the same project or program at different times or across 
different time spans can produce widely varying results, reflecting the uncertainty of research and the way in 
which the value of research outputs can change, depending on the context within which they exist - including 
subseguent advances in research. 

Measuring the impact of research is necessary and can be useful but it is important to use the results of such 
evaluations with care, recognising their fragility. In particular, it is important to assess impact in terms of the 
impacts the research aimed to achieve, not across all impacts which are possible. 
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Introduction 


The recent economic downturn has revived debate on the impact of research. A climate of increasing financial 
stringency makes it imperative to demonstrate that government investments in research are effective and 
providing value for money. 

In an environment of increasing and competing demands on funding, decision makers need to know that 
their support of research is contributing to the wellbeing of all Australians in concrete and demonstrable ways; 
and that that this investment is providing benefits greater than those that might arise from spending similar 
amounts in other areas. This is especially so as providing additional resources for competing programs can 
contribute (sometimes in more obvious and direct ways) to some of the outcomes research aims to achieve. 

As one obvious example, additional funding for the Pharmaceutical Benefits Scheme, for medical training or for 
hospitals might well have a more rapid (if shorter term and in the long run less significant) impact on health 
outcomes than increasing research funding. 

An additional reason for studying the impact of research is that this can complement the work of the ERA 
initiative, which measures the academic excellence of research. While there is evidence that academic 
excellence contributes to achieving impact (for example, as shown by the analysis of academic papers cited in 
patent documents), there is a view that the absence of readily available measures of impact can downplay the 
importance of some research that aims to achieve direct impact, especially when such research does not result 
in publications in high impact journals. 

More generally, there is a perception that Australia is not benefiting from its research investments to the 
extent to which it could and should. Measurement of research impact might provide data that will facilitate 
improvements in the operation of the national innovation system, in part by providing an improved 
understanding of the means through which research has impact and of how researchers can better use the 
impact pathways available to them. 

While the demand for increasing information on the impact of research is reasonable, supplying the 
necessary data is neither simple nor easy. The creation, transfer and subsequent application of knowledge 
and the success and effect of this are difficult to measure. This is because the relationship between research, 
knowledge transfer, application and the economic, social and other outcomes these can produce is complex. 
This backgrounder aims at exploring some of these issues and compares existing approaches in order to 
facilitate wider debate. 


Measuring impact is difficult 

Discussions about the impact of research and how to measure it are even more problematic than discussions 
about excellence and what we mean by it. There are many reasons for this. Research can have many different 
types of impact and the routes through which it achieves these impacts are diverse, often convoluted and 
not at all transparent. The time necessary to achieve an impact can be significant (even beyond the life span 
of individual researchers) and in some cases the impact may not be direct. Moreover, the apparent impact of 
a piece of research in the short term can be very different from its impact in the longer term - as shown by 
examples as diverse as Gregor Mendel in genetics or Alfred Wegner with continental drift. 

Impact can be negative or indirect 

While the focus of impact measurement tends to be on the beneficial effects of research, it is not unknown 
for research to create unintended or unwanted impacts that we need to balance against any obvious 
benefits. (The impact of chlorofluorocarbons on the ozone layer, of lead in petrol or the potential health 
and environmental impacts of some pesticides provide obvious examples.) Moreover, in many cases the 
benefits that flow from research can take the form of preventing harm or reducing damage, rather than of 
making things better. This is the case for much health research (disease prevention, more productive lives), 
environmental research (including research into natural hazards) and social science research (e.g. crime 
prevention). While it is not possible to measure directly the economic consequences of many of these impacts, 
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it is important to recognise and value them. Other research has impact by identifying problems about which 
we might not otherwise be aware and to which we need to respond - as with anthropogenic climate change 
or the hole in the ozone layer. 

Impact depends on players outside the research system 

An equally fundamental issue is that the researchers themselves might have had no idea of the potential 
benefits (or harm) of their research or its longer term implications. The impact that research has will often 
depend on the imagination, creativity and identified needs or problems of people outside the research system. 
This does not downgrade the importance of the research but it does raise questions about the use of potential 
impact as a means of selecting research proposals for funding. Consumers are often more imaginative and 
creative than researchers (and even better than business) in finding applications for new technologies. 

Perhaps one of the most difficult problems in trying to measure the impact of research is that in most cases, 
even when the research aims to achieve it, impact is the result of actions by parties other than the researchers 
themselves. Converting research outputs to innovations requires significant investment (intellectual as well 
as financial) and the bringing together of many complementary skills. This means that the cost of research is 
often a very small proportion of the cost of the investment necessary to produce the impact. This can create 
problems in trying to determine what proportion of the impact is due to research and what proportion to the 
other parties involved - who may have been able to purchase equivalent research outputs from elsewhere. 
Moreover, these other parties are using their intellectual as well as financial and other resources in producing 
public or commercial value from the research. 

A further consequence of the path to impact involving many and diverse players can be that research having 
the potential for considerable impact may have no effect because of inaction or a lack of complementary 
factors within the broader innovation system. More broadly, the willingness of consumers to take risks and 
use new technologies can play a major role in determining whether research produces the benefits it has 
the potential to produce. As a simple and very direct example, the benefits of a vaccine depend not just on 
whether it works but on the extent to which the target populations agree to vaccination. The same research 
can have quite different impacts depending on its cultural context and on the size of the local market for 
the benefits it might offer. Among other things this can mean that at least some of the benefits of research, 
especially research not protected by strong IP rights, can occur in countries other than that in which the 
research took place. 

All research builds on earlier research 

There can also be the problem that making effective use of the advances in one area of research might depend 
on complementary progress in other areas of research or technology. For example, the major benefits of a 
diagnostic test for early Alzheimer's will depend on other research having developed a useful response to 
such diagnosis, notwithstanding that a diagnostic might allow for studies of the early stage development of 
the disease that themselves help lead to a useful intervention. (In a similar vein, it is interesting to speculate 
what Leonardo de Vinci might have been able to achieve if he had had access to the materials and other 
technologies we have available today.) 

Equally important is that any research project builds and is dependent on all the previous research that 
preceded it - even the best of researchers stand not just on the shoulders of giants but on the great mass 
of researchers around the world who built the understanding and techniques that make current research 
possible. 

Recent attempts to apply real options analysis (developed in the financial services sector) to research 
evaluation demonstrate that research creates options that have a significant economic value even if society 
never takes them up; and it is important to recognise the insurance potential of the capabilities that research 
creates, in that they provide a capacity to respond to unexpected events, challenges and opportunities . 1 

1 See, for example www.csiro.au/files/files/pa6s.pdf 
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Research aiming to have impact is usually high risk 

Another important consideration is that research is often a high risk activity. In the normal course of things 
much research will fail. In the case of research whose sole aim is to advance knowledge, the failure to 
achieve the expected (or hoped-for) results still advances knowledge - even if the outcome is less exciting 
or interesting than that for which the researchers were aiming. However, for research aiming at defined 
commercial outcomes, failure is significantly more likely than success. 2 

Empirical analysis of business research and the experience of venture capital companies both demonstrate that 
only a very small proportion of projects succeed in commercial terms. Assessments of research impact need 
to factor in these high failure rates if they are to provide realistic and reasonable assessments of what research 
investments are achieving. The high risk involved in any individual research project also raises issues about 
what level of analysis is appropriate - whether it should be the project, particular types of research activity, the 
institution, the program or even total national effort. 3 

There can be different perspectives about the worth of an impact 

A further complication in this area is that discussions of research impact, like discussions of excellence, often 
have a strong emotional underpinning, reflecting value positions that while deeply held need not be explicit 
or apparent - even to those holding them. There can be a tendency to value certain kinds of impact more than 
others in a way that does not necessarily reflect their overall significance to national wellbeing or economic 
development. A direct financial return is concrete and relatively easy to measure but its overall economic 
impact may be less than that of an intangible research outcome such as improved national reputation or social 
cohesion. 

One complicating factor that runs through all impact assessment work is that there can be different 
perspectives as to whether a particular impact is positive or negative - and this can again depend on the time 
scale of the analysis. The impact of climate change research that flows from research identifying a problem 
provides an instructive if complex example but is not unique, even in much simpler situations. 

Environmental research that leads to the closure of a fishery might have an immediate negative economic 
impact, even though in the much longer term it will preserve a resource that might again become available 
for use. The fishing industry and conservationists might have very different views as to the nature of the initial 
impact - some of which may depend on their view about the excellence of the research and its disinterested 
nature. Similarly, research that suggests the need to cull kangaroos to preserve biodiversity and conservation 
values might receive a very different response from ecologists, conservationists and animal rights activists. 
Issues relating to water management can have many rival stakeholders each with their own, sometimes 
competing, value systems, who judge the outcomes of the same research, and its potential application, in very 
different ways. A program to reduce obesity in children might be expensive to implement, not have any serious 
economic benefits for 30 or 40 years, pose complex issues for the food industry and add to the cost of food. 

Assessing impact has to acknowledge alternative strategies 

A further problem in trying to assess the impact of particular research projects is to define an appropriate 
counterfactual. In a competitive research environment, with many teams working in the same areas of interest, 
the failure of a particular research project might delay a sought-for outcome but another team is likely to get 
there eventually (and perhaps sooner rather than later). In a complex world it is unlikely that there are single 
solutions to any problems and in the absence of a particular research project, another - even one adopting a 
completely different approach - will eventually provide a solution. 


2 Failure can be technical (e.g. problems with scale up), reflect cost over runs, result from a misreading of the market or have many 
other causes that do not present risks for basic research. 

3 Issues of risk are particularly important because the high risk of research provides one of the arguments for government support. 
If the risk is low, the end-users should normally fund the research themselves. 
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More difficult is the question of opportunity cost. Even when a particular project has produced an outcome 
that provided identifiable benefits, it is important to ask whether the same resources invested in a different 
project - even one directed at a different problem - would have provided even greater benefits. When 
government has provided the funding for research it becomes necessary to consider whether not collecting 
the money in the first place (by lowering tax rates) or spending the money on some other area of government 
might have produced a higher economic return . 4 

Measuring impact can distort behaviour 

It is also important to appreciate that identifying and using particular metrics can itself distort behaviour 
and lead to unintended outcomes, especially if such metrics feed into funding allocation systems. A focus on 
patenting, for example, can encourage institutions to develop extensive patent portfolios that cost more to 
maintain than the licensing revenue they produce. Moreover, patenting, which requires the public disclosure of 
an invention, can inhibit small businesses taking up an invention if their business IP strategy depends on trade 
secrecy - and this is often appropriate for SMEs given the cost of monitoring to identify infringement and the 
often prohibitive costs of trying to enforce IP rights. 

Just as insidious is that the areas of impact not subject to measurement will inevitably be seen as less 
important and not receive the attention they should. Given that some of the more pervasive and potentially far 
reaching impacts of research are often among the more difficult to measure, at least in the short term, this can 
in itself have negative consequences for the whole innovation system. 

What this means 

The main conclusions from these considerations are that identifying and quantifying the impact of research is 
difficult, that any estimate is at best a very rough guide and that the results of any assessment can vary widely, 
depending on the time frame used to allow for impact. An evaluation of Faraday's work on electromagnetism 
at the time he conducted his research, for example, would lead to very different conclusions from those that 
we might reach today in a world that could not operate or support its present population without electricity, 
electronics and photonics. While it is clearly necessary for accountability and other reasons to measure and 
evaluate the impact that research is having, the interpretation of such measurements has to recognise the 
ambiguities and uncertainties involved. 


Types of impact 

In order to assess research impact, it is necessary to agree on what form this impact can take. Understanding 
the areas in which research can have impact can help in developing a framework for identifying possible ways 
to provide measures (even if only partial measures) of research impact. 

In focussing on the impact that the outputs of the research can have, it is important to recognise the benefits 
that arise from the process of research - in particular, the role that research plays in developing the skills, 
knowledge, values and culture of the people - including postgraduate students - performing the research. 

A further impact of research is that it can improve or create the infrastructure (both tangible and intangible) 
on which further research can depend. Neither should it be forgotten that an important impact of research is 
that it creates an awareness of research performed overseas and a capability to make use of it, as well as the 
capacity to provide non-research scientific, technical and other consultancy services. Research also creates 
networks of personal contacts and of groups having similar or related interests. Together, all these impacts 
work to create a national capability that governments and others can draw upon as they need - they provide 
a kind of national insurance by providing a broad set of capabilities that is available as necessary to respond to 
unexpected events. 


4 An assessment of the cost-benefit of government research funding needs to take into account not just the funds the government 
provided but also the cost of raising these funds through the tax system and of delivering them to the recipients. 


Go8 BACKGROUNDER 23 | MEASURING THE IMPACT OF RESEARCH - THE CONTEXT FOR METRIC DEVELOPMENT 


PAGE 7 



One factor that can influence impact measurement is that some paths to impact are more visible and 
provide easier measures than others. Counting the number of patents granted, licence agreements, or the 
income received from licensing agreements or other contracts is relatively easy. It is much more difficult 
to demonstrate that a book presenting the results of research directly to the public has had an impact, for 
example by improving health and lifestyle, developing a greater degree of social tolerance or reducing energy 
consumption or waste. There are so many other factors influencing personal behaviour that it is difficult to 
isolate the role played by any single one - even though the potential benefits from explaining how and why 
to adopt a healthier and more sustainable lifestyle, or the benefits that arise from a more open, tolerant and 
multicultural society, may be just as great as those that arise from the commercialisation of a widget. This is 
one reason that impact studies tend to focus more on technology and less on conceptual developments and 
theoretical insights. 

The following discussion categorises the various forms of impact that research can produce. Different 
measurement technigues and indicators are necessary to deal with the different categories of impact. 

However, it is also necessary to acknowledge that any particular research project (or even program or funding 
program) may be aiming at particular kinds of impact. Moreover, research in different disciplines provides the 
potential for creating quite different kinds of impact, so that any framework for measuring impact has to take 
into account the differences between disciplines. 

It is necessary to assess a project (or program or institution) in terms of the kind or kinds of impact it was 
aiming to produce (and capable of producing), not to assess it across the possible range of impacts that 
research in general can produce. To do otherwise would ignore the division of labour that exists and is 
necessary, within the national research system and promote homogenisation rather than differentiation and 
specialisation that can improve the operations of the system. This is not to say that serendipitous outcomes do 
not occur and it does become important to capture them when they do; but, for example, to assess an optical 
astronomy program in terms of its contribution to the competitiveness of Australia's manufacturing industry 
would not make sense. 

More effective teaching 

Within a university research has close links with teaching and the two influence each other in many ways. Two 
are of particular importance. First and most obvious is that learning within a research environment exposes 
students to the most current and up to date knowledge, concepts and developments before they move into 
the wider workforce. Exposure to researchers brings students into contact with leading edge thinking, to 
understanding that has not yet reached the text books and to ideas which have yet to find application. 

Perhaps even more important is that learning within a research environment generates a culture which 
challenges existing ideas, accepts that knowledge is provisional and which respects alternative points of view 
until the available evidence permits discrimination. This kind of culture, which students then take with them 
as they move into business, government or other working environments, is one that facilitates innovation 
and creates a preparedness to challenge existing practices and to search systematically for ways of doing 
things better. 

Research produces other benefits for teaching, as well. For example, teaching in a research environment 
can provide students with access to new and advanced equipment, some of which business might not yet 
use, or to innovative techniques that have application outside research. The personal contacts and networks 
that result can also be important in helping to develop an innovative culture outside of universities and can 
potentially have far reaching effects - including in the application of research. 

Advances in knowledge 

For research to have impact it has to have effects that go beyond the immediate outputs of the research 
and which extend beyond those who performed the research. At the most basic level, all research advances 
knowledge. Some research has this as its primary, if not only, purpose. The improvements in understanding 
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and the increased sophistication in our ability to modify the world that result from improvements in our 
understanding, together form the most common and pervasive impacts of research. This provides a number 
of possibilities for assessing impact. One is to assess the advance in knowledge or understanding compared 
to what was known before the research; another is to examine how widely the researchers disseminated the 
improved knowledge to those having an interest in it, or who were capable of using it or of building on it to 
produce further advances; and a third is to assess the extent to which researchers or others made use of the 
advance in knowledge that the research produced. 

Mechanisms such as peer review and the numbers of publications, along with citation and other bibliometric 
measures, can provide partial indicators of impact in this area. The citations of research papers in patent 
documents can provide an indication of the extent to which research is extending its influence beyond 
academia and the broader research community. Certainly in the short and medium term, there need be no 
direct relationship between the extent to which the research advances knowledge and its impact in other 
areas. The most exciting and step change improvements in knowledge may at first have no direct or immediate 
practical significance. Research of a standard to win the Nobel Prize or Fields Medal does not always produce a 
direct, tangible improvement in national or global wellbeing. 

One direct impact of advancing knowledge can be to enhance the reputation of the responsible researchers. 

It is possible to assess this using proxy measures such as academic awards, election to learned academies, 
invitations to prestigious meetings, and so on. The high international reputation of individual researchers and 
institutions has an impact of national standing and on the attractiveness of a country as a place to learn, work 
and invest. 

Additional investment 

One measure of the potential impact of research is the willingness of other parties to invest further in the 
research and its further development. This is especially the case when those other investors come from the 
business sector. The preparedness of a business to invest in the development of a pilot plant or to otherwise 
investigate scale up, or of a drug company to pay for early stage clinical trials, provides a clear demonstration 
that the research had relevance to those new investors and that they recognise that it has potential for impact. 

Financial return 

One of the most sought-after impacts at an institutional level is often financial return. If the organisation 
performing the research is able to sell or licence the research outputs and receives more than the full cost 
of the research, it is likely to conclude that the research has had a positive impact. The proximate users of 
the research outputs might use a similar measure. However, a financial return does not capture the benefits 
flowing to the ultimate consumers, including any consumer surplus and broader spinoffs. While financial return 
is sometimes easy to measure, it has many limitations. Many areas of publicly supported research will not 
produce outputs that can lead to direct financial returns and in any case the financial return will normally be a 
gross underestimate of the true value and impact of research. 

Economic impact 

From a national perspective the economic impact of research is much more important than measures of 
direct financial return. Not least this is because in the case of government funded research any financial 
return to the government is indirect, acquired through the tax system (flowing from increased employment, 
competitiveness, domestic sales and exports) or from the reduced expenditure that results from increases in 
the efficiency and effectiveness of government service delivery. In most cases these are not easily attributable 
to any particular project. 

Measuring economic impact is a much more difficult job than measuring a direct financial return and is much 
more uncertain, given the various assumptions that may be necessary and the need to take into account 
multiplier effects. Nevertheless, it is possible to make estimates at an aggregate level of economic impact and 
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examine the contribution of research to economic growth and productivity, as well as to examine the impact 
of research on such measures as business start-ups, exports or measures of national wellbeing. 

Estimates of economic impact can be useful for some purposes and in theory can provide a more realistic 
estimate of the overall impact of research than more limited analysis. However, it is worth noting that a 
comprehensive study by the Productivity Commission, using the best available data to conduct a detailed 
econometric analysis of the impact of Australia's research and development on productivity concluded that: 

A major message from all the analysis is that, at least for the time being, empirical estimates of the effects 
ofR&D on Australian productivity are unreliable. Any assessment therefore requires a high degree of judgment . 5 

Social impact 

Many areas of research have an impact on society as a whole, for example by providing better health outcomes 
or increased safety and security. Research can provide knowledge and understanding that can help inform 
policy development or which can lead to the improved efficiency and effectiveness of public service program 
delivery and development. For example, research can support the development of regulatory systems which 
can affect everything from road rules to the availability of alternative medicines and even new voting systems. 

A major problem with measuring social impact is that the routes through which research can influence 
individual behaviour or inform social policy are often very diffuse. Change often requires action on multiple 
fronts and, as in the case of commercialising 'hard' technology, requires the participation of actors beyond 
the researchers. The role played by research can be one of creating a climate facilitating change, as well as 
providing particular avenues and directions for change. 

In a country such as Australia, governments look for or need a democratic mandate to take action. This 
requires that an issue be on the public agenda and that the public understand and preferably support the 
actions the government intends to take with respect to the issue. Research can play an important role in 
both these areas, as well as by providing rigorous analysis of the implications and broader effects of the 
options available to a government for dealing with an issue. Scholarly books and reviews can help create the 
intellectual climate within which decision makers and policy advisors operate. 

Equally important is creating public interest and awareness by adopting the role of a public intellectual, 
communicating through the media and using other non-scholarly forms of publication. However, such 
activities are difficult to measure and it may be impossible to demonstrate a direct link with the policy 
decisions made and the outcomes of these decisions. Nevertheless, they are all important paths to the use 
of the research, along with other activities such as consultancy work for government departments, preparing 
submissions for different government inquiry processes or sitting on advisory and other committees and even 
social interaction with politicians and bureaucrats. 

Environmental impact 

There is often a more direct link between environmental research and improved environmental outcomes 
than is the case for many social impacts of research. Nevertheless, some kinds of environmental research face 
the same difficulty in demonstrating direct links between research and impact as can be found in assessing 
social impacts. Climate change provides an obvious example and raises the additional point that research 
can be important, add value and have impact by identifying problems of which we might otherwise not be 
aware. This is quite apart from the role research can play in generating possible solutions to the problems 
which it has identified. As another example, the concept of environmental services and the understanding that 
natural ecosystems provide services such as water purification, nutrient and mineral recycling, or pollination 
that are of immense economic value has improved decision making with respect to the management of 
land, atmospheric, freshwater and marine resources, although in many cases it would be very difficult to 
demonstrate a direct impact and measure its economic return. 


5 www.pc.gov.au/ data/assets/pdf_file/0004/37183/economicmodel ling. pdf 
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Intangible impacts 

While economic, social and environmental benefits clearly add to national wellbeing, there are other less 
tangible benefits of research that may be difficult to measure but just as important. Research in areas such 
as history, archaeology and the biology of indigenous plants and animals can add to a sense of national 
and cultural identify. Internationally recognised research across any field adds to national reputation, can 
be an important factor in gaining Australia a seat at the international table and demonstrates that Australia 
is a nation contributing to the study of global problems and playing its part in seeking problems to them. 
Australia's reputation as a leading nation in radioastronomy has the potential to make Australia the location 
for the SKA, a $2 billion international research facility with the prospect of considerable spin-off benefits 
for Australia. 

In a very real sense, international research is a 'trading network' in which nations are able to access the 
knowledge and technology produced by others according to the level at which they are contributing. Research 
of a guality to bring Australia into this network has a considerable value above and beyond the benefits of any 
other impacts it might have. Moreover, having a reputation that flows from internationally recognised research 
strengths is important in attracting to Australia the very best researchers from overseas, helps facilitate 
international collaborations and is a factor that multinational corporations take into account when considering 
where to invest. In a more practical sense, having a strong domestic research base is also necessary to keep 
abreast of international research and preserves the capability to understand the significance of such research 
and make use of it. 

Methods of measuring impact 

As already mentioned, there are many possible ways to measure the impact of research, although none 
can provide unambiguous or unarguable results. Moreover, evaluation of the same project, program or 
institution at different times or over different time spans can lead to widely divergent conclusions as radical 
innovations are rare, uncertain and unpredictable. The various assessment methods operate at different levels 
of aggregation, from that of individual projects to the overall impact of total national investment in research. 
Some are retrospective, some are prospective and some can be either. 

These different methods serve different purposes and it is necessary to choose a method or methods that 
best relate to the purpose the assessment exercise is to serve. In practice it is usually necessary to use a variety 
of methods operating at different levels to provide useful and reliable information on the impact that the 
research has had. No single method is able to provide a complete picture. Nevertheless, each of these methods 
can contribute to our understanding of the relationship between research and its economic, social and 
environmental outcomes, as long as we recognise its limitations. 

While different methods produce data having different degrees of robustness, this does not alter the fact that 
different approaches can appeal to different audiences. An econometric analysis that excites Treasury is not 
necessarily the best way to convince the general public that governments should fund research. Exciting and 
unusual anecdotes that attract media interest can help demonstrate the importance of research to the broader 
community but lack the rigour that many decision makers reguire. Sectoral case studies and cost-benefit 
analyses might interest bureaucrats in line departments but officials from the Department of Finance might 
give them less credence on the grounds that they are selective and tendentious. Any strategy to communicate 
the importance of research to economic development and broader national wellbeing has to use a diversity of 
approaches selected to meet the particular needs of individual target groups. 

Input measures 

While by definition input measures do not identify impact, they can be useful in benchmarking exercises and 
in setting the boundaries for the kinds of impact one might expect at a national level. Important indicators can 
include the proportion of national effort going into basic, applied and experimental development research; 
the proportion of research performed in different research fields or directed towards different socio-economic 
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objectives; and the proportion of research effort directed towards agreed national priority areas. In all cases the 
trend data can add valuable information. Another important input measure can be the source of funding - for 
example, business funding or grants requiring end-user participation may (or should) be more likely to lead to 
immediate impacts than funding from government programs aimed at supporting universities. 

Input measures can demonstrate the way in which research institutions and program delivery agencies are 
responding to the government's national priorities, and are aiming to achieve impact in areas the government 
has deemed as important. Similarly, figures showing trends in the value of commercial contracts received by a 
university can indicate the extent to which it has developed capabilities that the private sector has identified as 
being important enough to invest in. Other measures of cross sectoral collaboration and interaction, including 
partnerships, might also provide proxy input indicators of (at least potential) impact. 

A more intangible input measure, but one which can significantly affect the likelihood of research having 
a positive impact, is the nature of the research management process. Sophisticated research management 
involving fast failure approaches and the continual evaluation of research according to criteria that relate to 
the excellence of the science, its continuing relevance to the intended objective and changes in the path to 
impact environment will have a greater propensity to achieve impact than research managed in a different 
way or left to the intellectual curiosity of an individual researcher. However, the purpose of research has 
to determine the appropriate research management style and what makes best practice for experimental 
development is by no means best practice for basic research. 

Output measures and benchmarking 

Output measures can help in assessments of research productivity, but only if placed in the context of the 
intent of the research and the objectives of its funding body. The immediate outputs of research are clearly one 
of the stages on the path to impact. Publications and then citations can demonstrate the extent to which the 
research is having or has had influence. However, while assessments of academic research excellence might 
focus on the most academically prestigious journals, measures of research impact might need to consider 
other publications - including client reports, technical manuals, patents, text books, the media, newsletters, 
submissions to government inquiries and communications directed towards the general community or to 
particular industry sectors, and so on. 

Other output measures can include the use of processes that have the potential to lead to impact. These can 
encompass the use of technology transfer mechanisms such as industry seminars, industry secondments (in 
either direction), field days which demonstrate new techniques and approaches, participation in government 
committees and policy development processes, participation in industry and academic meetings and seminars, 
preparing popular publications, research consultancy work and the provision of other advisory or technical 
services. While none of these is an impact in the narrow sense, they all indicate an intent to have the research 
used and a preparedness to take action that will bring the research to the attention of those with an ability to 
use it. In many cases these kind of mechanism operate at a level above that of individual projects and provide 
access to the knowledge, experience and capabilities that have accumulated from research over many years. 

Expert review 

Just as peer review can be useful in assessing the quality of academic work in an academic context, expert 
panels with relevant experience in different areas of potential impact can be useful in assessing the difference 
that research has made. Such panels can also provide useful comment on the extent to which researchers are 
making effective use of the mechanisms that have the potential to lead to impact. 

As with peer review there can be issues related to the subjectivity of the judgements made and it is important 
to ensure that the experts have the experience necessary to assess the particular impacts the research claims. 
Despite these limitations, the use of expert panels can provide informative evaluations, especially when 
comparing claims from different institutions or groups. Expert panels are often useful in conjunction with some 
of the other methods that are available and can sometimes help temper their shortcomings. 
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Anecdotes 


Many researchers can provide narrative accounts of the economic or other benefits that their research projects 
have produced. Such anecdotes do not rely on detailed analysis but simply describe how a particular research 
project contributed to the development of an identifiable outcome that extended beyond the research 
community. Anecdotal evidence is often personal and subjective but does demonstrate an awareness among 
the researchers of the ways in which their research can have impact - which is important in itself. While a long 
list of outcomes based on anecdotal accounts can appear impressive, it does not provide any quantitative 
indication of the level of economic benefits or of what level of additional investment was necessary to achieve 
them. However, it does identify some of the economic benefits that research is producing and can help 
illustrate their variety. 

Case studies 

Case studies use the detailed analysis (to varying degrees of rigour) of individual projects to explore the 
ways in which research produced an economic or other outcome that contributed to national wellbeing. The 
preparation of case studies is often a research project in its own right and involves people other than those 
working on the research under study. While case studies provide data at the micro-level and the impacts 
they identify relate only to the particular project or program they examine, case studies do have many 
advantages. In particular, they can help identify the players involved beyond the research performers and can 
help in estimating the total costs of converting an invention into an innovation. Case studies are necessarily 
retrospective and will usually draw on anecdote as well as data, not least because of the complexity of the 
innovation process and the diverse array of players usually involved. However, they can play an important role 
in developing a better understanding of innovation processes. 

Cost-benefit analysis 

Cost-benefit analyses, like case studies, normally start with particular research projects or programs. They are 
often less rich in detail, although richer in quantitative data, than case studies. The objective of a cost-benefit 
analysis is to identify the full costs of the research and to assess in detail the economic value of the benefits 
that have flowed from the research or to estimate the likely benefits that will flow from current investments. A 
single study can be expensive and this limits the extent to which it is possible to use them. 

The quality of cost-benefit analysis can vary enormously. For example, some analyses take into account only 
the cost of the research and ignore the often greater financial investments that come from those using or 
commercialising the research. There can also be significant differences in the extent to which the analysis takes 
into account the full range of benefits that flow from the research or concentrate on the direct commercial 
outcomes. The assumptions made in conducting the analysis (including the details of the counterfactual) 
can make a big difference to the conclusions, especially when looking at projected rather than retrospective 
benefits. There may also be problems in attributing costs or benefits to individual projects, given the many 
interdependencies that inevitably exist, as hindsight studies make clear. 

Most cost-benefit analysis examines carefully defined projects or small programs but some studies take a 
broader approach, as when they examine the return on the research investments made by whole funding 
programs or by institutions. However, the broader the study, the more courageous the assumptions the 
analysts have to make. Collating the data necessary for even a single project can be difficult and time 
consuming. For this reason a broad survey tends to combine an unsophisticated (even simplistic) cost-benefit 
analysis with an anecdotal approach. This kind of analysis can often fall into the category of 'advocacy research', 
in that it collects data and conducts analysis to support an already agreed position. 

As an example of what cost-benefit analysis can achieve, a major study of the return on investment in ARC 
funded research examined the benefits it could trace from ARC research funding. The study used the following 
categories of benefit: 
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• building the basic knowledge stock; 

• the generation of commercialisable intellectual property; 

• improving the skills base; 

• improving access to international research and international networks; 

• better informed policy making; and 

• health, environmental, social and cultural benefits. 

The study examined links between ARC funding and each category of benefits, past and current and estimated 
the total social rate of return (the permanent increase in GDP as a percentage of the dollar cost of the 
investment that lead to this increase) on ARC investment in Australia at 39 per cent. This is high, considering 
that the average social rate of return on publicly funded R&D appears to be around 25 per cent. The authors 
noted in addition that this estimate of returns from ARC funded R&D is only a partial measure of overall 
benefits, as it was not possible to quantify some benefits (such as health, environmental, social and cultural 
benefits). Overall, the study (commissioned by the ARC) concluded that it is highly plausible that ARC funding 
generates higher returns than is the average for all publicly funded R&D. 6 7 

Hindsight studies 

Whereas case studies start with research projects and work forwards to map their impact, hindsight surveys 
start with an impact and work backwards to identify what contributions research made to the impact. A case 
study may show that a single area of research contributed to many different impacts, a hindsight survey might 
identify that many different areas of research were necessary to achieve particular impacts. As with all such 
studies there can be a high degree of subjectivity and the findings can depend on the interest, knowledge and 
perspectives of those conducting the study. Moreover, a rigorous hindsight survey can be very expensive and 
time consuming. 

The first hindsight study was probably that performed by the US Office of the Director of Defense Research 
and Engineering in 1 9697 Th is analysed twenty weapon-systems and major military equipment to identify 
applications of science and technology not used in predecessor military systems designed to meet roughly the 
same requirements. The study traced back the evolution of the new technology represented in each system to 
critical points called "research or exploratory development (RXD) events." Specialist teams identified 71 0 such 
events. The analysis separated the basic research performed to solve a specific assigned problem (directed 
basic research) from the basic research performed to expand the frontiers of scientific knowledge (undirected 
basic research). The analysis found that RXD Events from directed research emerged in systems development 
approximately nine years following their conception, while it took 20 or more years for some events from the 
undirected category to have an impact. More importantly, the study noted that 95 percent of the knowledge 
used in these events was an outcome of activities supported either directly or indirectly by the Department 
of Defense; and that 61 percent of the RXD Events identified had a specific systems' requirement as a research 
objective and that over 85 percent of the technological events occurred after a problem applications group 
had defined the problem they needed solved. 

Even before Project Hindsight had produced its final report, university researchers responded to these 
conclusions by noting that because Project Hindsight looked back only 20 years, it had missed the basic 
research that provided the underlying foundation for the technological developments it had studied. To 
redress this, the National Science Foundation commissioned a study {Technology in Retrospect and Critical Events 
in Science) which performed a hindsight analysis on five major technological developments (ranging from the 
contraceptive pill to the electron microscope) and looked back 50 years. 8 This concluded that 70 per cent of 
the 340 events the study identified were the result of non-mission (basic undirected) research, with 20 per cent 

6 www.arc.gov.au/pdf/ARC_wealth_of_knowledge.pdf 

7 www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=AD0642400 

8 Illinois Institute ofTechnology, Research Institute. 1 968. Technology in Retrospect and Critical Events in Science. Chicago: Illinois 
Institute ofTechnology. 
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originating from mission research. Furthermore, universities had performed around 75 per cent of the non- 
mission research and a third of the mission research. 

In considering the difficulties posed by hindsight studies, it is worth reflecting that foresight studies aiming to 
predict future technological developments or even advances in science are even more problematic. Even the 
best informed experts are likely to miss some of the most momentous impacts of research that may occur over 
the coming years because research - and the impacts it will have - are by their very nature unpredictable and 
depend in part on chance and the concatenation of circumstance. Many of the most significant and pervasive 
outcomes of research were not the result of deliberate planning but an outcome of the accidental interplay of 
curiosity, imagination and opportunity. 

Surveys 

Case studies and cost-benefit analyses are necessarily selective and local, in that it is necessary to choose 
particular examples for study. It can be difficult to extrapolate any conclusions beyond the particular project 
or organisation under study. In contrast, surveys can provide a more comprehensive and more balanced set of 
data. They can provide a means of gathering comparative data, especially but not exclusively data that is useful 
in comparing different institutions or sometimes programs. However, by themselves surveys usually address 
only part of the impact eguation and while they can collect data that it is possible to use for guantitative 
comparisons, in some cases this may be subjective (as with stakeholder surveys). 

Stakeholder surveys 

Some forms of survey may gather a type of anecdotal information, but from a user rather than researcher 
perspective. For example, it is possible to survey the customers and other stakeholders of research-performing 
departments or of technology transfer offices to assess stakeholder views of the importance of the research 
and the ease of working with the researchers or research office to create impact. The responses will usually 
be gualitative, anecdotal and unsophisticated. Nevertheless, stakeholder perceptions of impact and the 
willingness of a university to work with outside agencies to achieve impact can be valuable in a political 
context. Such surveys can also help identify ways of increasing impact by improving linkages, changing 
perceptions and removing impediments to the flow, exchange and use of knowledge. Trends identified in 
stakeholder surveys can also help indicate changes in the willingness of an organisation to respond to the 
possibilities for achieving impact. 

It is necessary to take care in interpreting stakeholder surveys because of the conflicting views that can exist 
about the proper role and responsibilities of the organisations involved. For example, business expectations of 
universities can be unrealistic and ignore the broader roles of universities in serving the whole community and 
in doing research that complements business research, rather than substitutes for it. A survey seeking general 
comments from business on university research might have a very different response from a survey seeking 
comment on the contract research a university performed for business. 

Commercialisation surveys 

While stakeholder surveys collect subjective and anecdotal data, commercialisation surveys normally collect 
quantitative data relating to matters such as the number of staff devoted to technology transfer, spin-off 
companies, patents and other IP rights applied for or granted, and licensing income. They may include 
customer data, showing the number of customers and the size of contracts. These surveys can provide a 
comparable set of data from different organisations. 9 

While commercialisation surveys provide quantitative data that can be useful for comparisons, benchmarking 
and monitoring trends, it is still necessary to exercise care in the interpretation of such data. The potential 


9 The most recent Australian National survey of Research commercialisation is at: www.innovation.gov.au/lnnovation/ 
ReportsandStudies/Pages/NationalSurveyofResearchCommercialisation.aspx 
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for narrowly defined commercialisation activity will vary depending on the discipline mix of an institution, 
its balance between different types of research activity and its particular roles, responsibilities and strategies. 
Moreover, commercialisation as measured by surveys provides a very narrow perspective on research impact, 
even in the confined context of achieving impact through business. 

Commercialisation surveys generally do not examine the cost side of the equation and either ignore (or do 
not assess) the actual economic impact of each organisation's activities. (In most cases the economic impact 
of a technology would far exceed the licensing fees going to the inventor, or even the improved financial 
performance of the firm using a technology.) Maintaining IP rights, for example, can impose a significant cost 
if there is no customer interested in using those rights. These surveys also miss the economic impacts that can 
be more difficult to measure, such as those relating to the impact of the research on skills, policy advice or the 
very many uses of research that occur without the need for explicit commercialisation vehicles. 

An important issue in mandating such surveys as measures of institutional performance is that they can 
influence the behaviour of organisations covered by the survey such that they favour strategies that increase 
the financial return to the organisation as distinct from those which increase the economic return to the 
nation. For example, using commercialisation surveys to measure performance can encourage patenting and 
the granting of exclusive licences in situations when the best return for the nation might come from the rapid 
and free diffusion of the technology to those able to use it. Patenting a research method, while producing a 
revenue stream for the patent owner, might decrease impact by restricting the use of the technique. 

Commercialisation surveys can also encourage the formation of spin-off companies that face all the attendant 
risks of a new business when the most certain path to market might be through an existing business. This 
can be of particular significance for organisations performing research in sectors such as agriculture or in 
public good areas such as environmental management. This raises interesting policy issues relating to the 
responsibilities that go with the use of public research funds, quite apart from the implications of promoting 
strategies that have the potential to reduce impact. A further problem is that using performance indicators 
such as patent applications (or even grants) can encourage patenting even when the technology patented 
may be of little if any commercial value. Institutions can end up spending more on patent fees than they 
receive in licensing or other revenue flowing from their patents. 

Economic models 

The development of conceptual and theoretical models can help explain why research is important and 
provide an understanding of the means through which research can lead to improved productivity or 
economic growth. They provide intellectual support for a relationship that many supporters of research take for 
granted. These models have value in themselves in providing potentially testable explanations and provide a 
means of examining the impact of research in quantitative terms. This is because econometric analysis can use 
empirical data to assess the impact of research expenditure on the economy using these models. 

Science and technology policy analysts have argued that research is important to economic growth for 
many years but it was not until the appearance of the 'new growth theories' heralded by Paul Romer's 1 986 
paper Increasing returns and long-run growth that these arguments achieved a level of respectability with the 
economic decision makers in government. This new theoretical framework stimulated a great deal of empirical 
work that has led to a much better understanding in economic terms of the impact that research investments 
can have. 

The most important characteristic of the new growth theories was that they recognised increasing returns 
to scale in production. In particular, the most important drivers for these increasing returns are the stock of 
knowledge generated by research and development; returns to investment in education and skills; and returns 
from learning by doing. These economic models tend to emphasise the non-rival nature of knowledge - that it 
is possible for one person to use knowledge without in any way detracting from the ability of someone else to 
use the same knowledge. This combines with the non-excludability of knowledge, the ability of anyone to take 
existing knowledge and build on it to create more knowledge which itself is non-rivalrous and non-excludable. 
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While such models do not in themselves provide a measure of the impact of research, they do provide part of 
the context within which governments make decisions about the funding of research and why government 
investment in research is worthwhile and responsible. They enable potentially testable explanations to replace 
assertion and provide a framework for econometric analysis. 

Econometric analysis 

Case studies, cost benefit analysis and surveys all provide data that can relate to individual organisations. 
Econometric analysis uses a range of numerical analysis techniques to explore the macro impacts of research. 
The coordinating departments (such as Treasury, Finance and Prime Minister and Cabinet) often favour 
econometric analysis because it operates at a national level and is not possible to manipulate its outcomes 
through the careful selection of examples, as with other some other methods. Even so, there are important 
consequences flowing from the assumptions made in developing the models and the robustness of the 
available data. 

Econometric analysis uses empirical data to explore the economy-wide consequences of changing investment 
in research and the economic impacts of research performed in different sectors (government, higher 
education, and private) rather than individual agencies. Such analysis can demonstrate a relationship between 
productivity growth and research, so helping to justify government funding of research. However, because 
it is operates at a macro level, econometric analysis does not help make decisions about where the research 
funding should go or about the performance of individual research agencies. On the other hand, because it 
does not need to identify the individual benefits but considers only the overall economic impact, econometric 
analysis might help capture those economic returns which can be very difficult to measure at lower scales 
of analysis. However, by its very nature, such analysis cannot capture some of the more intangible impacts 
of research or those that arise from reducing or preventing negative effects rather than facilitating positive 
outcomes. 

As an example of the use of econometric analysis, an influential OECD study investigated the long-term 
relationship between productivity growth and technical change. 10 Multifactor productivity (the residual after 
removing the contributions of labour and capital from GDP growth) provided the measure of productivity. 
Sources of technical change used in the analysis were business R&D, foreign R&D (business R&D performed in 
other OECD countries) and public R&D (including the government and higher education sectors). The study 
used data from 1 6 OECD countries from the period 1 980-98. Among the conclusions of the study was that: 

• The effect of government and university performed research on productivity is positive and significant, 
outweighing the cost of public research. 

• The effect of public research is larger in countries where universities (as opposed to government 
laboratories) have a higher share in public research. This may reflect government laboratory research 
addressing public missions that do not impact directly on productivity (defence, health, and environment) 
whereas universities provide industry with basic knowledge that provides the basis for technological 
innovation. (The paper also suggests that one reason for this is that universities receive a greater proportion 
of their funding through competitive processes.) 

More recently, the Productivity Commission in its detailed and useful study of public support for science and 
innovation in Australia, conducted a detailed and technical review of similar work from around the world. 

Their report provides a useful summary of Australian and international work. The Commission's summary of its 
analysis of econometric and other work aimed at assessing the impact of research was that: 

Taking account of multiple sources of evidence, there are likely to be significant aggregate economic, social and 
environmental benefits from publicly supported science and innovation, but guantitative estimates are unreliable. 


10 Dominique Guellec and Brunovan Pottelsberghe de la Potterie: (June 2001): R&D and productivity growth: panel data analysis of 1 6 
OECD countries. STI Working papers 2001/3 OECD DSTI/DOC (2001)3 


Go8 BACKGROUNDER 23 | MEASURING THE IMPACT OF RESEARCH - THE CONTEXT FOR METRIC DEVELOPMENT 


PAGE 17 



Conclusions 


There is no doubt that research has a major impact on our quality of life and on our general wellbeing; or that 
it achieves these impacts through a diversity of pathways and through indirect as well as direct routes. We can 
see this simply by looking around at the materials we use, the technologies available to us and the speed at 
which things change. 

The outcomes of research affect every aspect of our existence - whether work, social life or leisure. Everything 
from our health care and entertainment experiences to the structure of our economy, new businesses and 
communication opportunities are completely different now from what they were even 20 years ago. Research, 
the improved understanding it provides and the technologies it produces, impact on every aspect of our life 
and social relationships. 

While it is obvious that investment in research has impact and that many of these impacts make life better, 
it is much more difficult to demonstrate the contribution made by individual research projects or to quantify 
the contribution that research has made, especially given all the other inputs necessary to achieve successful 
innovation. This should not be surprising, recognising that many of the most important contributions of 
all research are systemic rather than linear. They do not exist in isolation. Their creation built on much that 
had gone before and their impact was the result of significant and complex contributions from outside the 
research system, as well as on a large element of chance. 

Even though it is difficult, it is still necessary to try to measure the impact that research has, partly to meet the 
general accountability that applies to all investments and especially to government funding, but also as an aid 
to making investment decisions in the first place and to improving the effectiveness of the national innovation 
system. At the same time, it is necessary to consider that by its very nature research is dealing with what we do 
not know, so that the significance of its outputs are to a greater or lesser degree unpredictable - which is why 
serendipity is always important. 

In attempting to evaluate the impact of research it is important to recognise the limitations that flow from 
the complex innovation system within which research operates, the varying pathways through which impact 
can arise and the different timelines within which different kinds of research might develop their value. It is 
necessary to recognise that different kinds of research aim to produce different kinds of impact and to assess 
research projects within their own boundaries of intent and the purpose for which they received funding. 

This requires a careful choice of the available methods and even more care in interpreting their results. Most 
importantly, it is necessary to recognise that an evaluation is at best tentative as subsequent research can have 
a major effect on the potential impact of previous research. 
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